Indirect agent connection improvements by sureshanaparti · Pull Request #13028 · apache/cloudstack

sureshanaparti · 2026-04-15T05:16:20Z

Description

This PR improves the Indirect agent connection handling, has the following improvements.

Enhances the Host connecting logic to avoid connecting storm (where Agent opens multiple sockets against Management Server).
Implements HostConnectProcess task where Host upon connection checks whether lock is available, traces Host connecting progress, status and timeout.
Introduces AgentConnectStatusCommand, where Host checks whether lock for the Host is available (i.e. "previous" connect process is finished).
Implementes logic to check whether Management Server has lock against Host (exposed MySQL DB lock presence via API)
Removes synchronization on Host disconnect process, double-disconnect logic in clustered Management Server environment, added early removal from ping map (in case of combination ping timeout delay + synchronized disconnect process the Agent Manager submits more disconnect requests)
Introduces parameterized connection and status check timeouts
Implements backoff algorithm abstraction - can be used either constant backoff timeout or exponential with jitter to wait between connection Host attempts to Management Server
Implements ServerAttache to be used on the Agent side of communication (similar to Attache on Management Server side)
Enhances/Adds logs significantly to Host Agent and Agent Manager logic to trace Host connecting and disconnecting process, including ids, names, context UUIDs and timings (how much time took overall initialization/deinitialization)
Adds logs to communication between Management Servers (PDU requests)
Adds DB indexes to improve search performance, uses IDEMPOTENT_ADD_INDEX for safer DB schema updates

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

sureshanaparti · 2026-04-15T05:17:43Z

@blueorangutan package

sureshanaparti · 2026-04-15T05:20:55Z

@blueorangutan package

blueorangutan · 2026-04-15T05:22:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-04-15T05:24:53Z

Codecov Report

❌ Patch coverage is 20.24892% with 1666 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.11%. Comparing base (96ca1b2) to head (e633b54).

Files with missing lines	Patch %	Lines
agent/src/main/java/com/cloud/agent/Agent.java	9.83%	320 Missing and 1 partial ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	39.30%	260 Missing and 21 partials ⚠️
...t/src/main/java/com/cloud/agent/ServerAttache.java	8.19%	266 Missing and 3 partials ⚠️
.../main/java/com/cloud/agent/HostConnectProcess.java	22.83%	168 Missing and 1 partial ⚠️
...c/main/java/com/cloud/utils/nio/NioConnection.java	27.06%	85 Missing and 12 partials ⚠️
...s/src/main/java/com/cloud/utils/nio/NioClient.java	18.36%	79 Missing and 1 partial ⚠️
...ils/backoff/impl/ExponentialWithJitterBackoff.java	0.00%	70 Missing ⚠️
...b/src/main/java/com/cloud/utils/db/GlobalLock.java	19.27%	49 Missing and 18 partials ⚠️
...main/java/com/cloud/agent/SynchronousListener.java	0.00%	47 Missing ⚠️
...rk/db/src/main/java/com/cloud/utils/db/DbUtil.java	20.51%	31 Missing ⚠️
... and 20 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13028      +/-   ##
============================================
+ Coverage     18.09%   18.11%   +0.02%     
- Complexity    16717    16764      +47     
============================================
  Files          6037     6047      +10     
  Lines        542546   544038    +1492     
  Branches      66431    66572     +141     
============================================
+ Hits          98148    98540     +392     
- Misses       433378   434429    +1051     
- Partials      11020    11069      +49

Flag	Coverage Δ
uitests	`3.52% <ø> (ø)`
unittests	`19.27% <20.24%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2026-04-15T06:23:00Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17497

sureshanaparti · 2026-04-15T06:33:54Z

@blueorangutan test

blueorangutan · 2026-04-15T06:36:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-04-15T07:03:15Z

[SF] Trillian Build Failed (tid-15880)

github-actions · 2026-04-30T03:29:40Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2026-05-06T19:46:06Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

borisstoyanov · 2026-05-07T09:59:09Z

@blueorangutan package

blueorangutan · 2026-05-07T10:00:04Z

@borisstoyanov a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-05-07T10:00:56Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17752

…ements. - Enhances the Host connecting logic to avoid connecting storm (where Agent opens multiple sockets against Management Server). - Implements HostConnectProcess task where Host upon connection checks whether lock is available, traces Host connecting progress, status and timeout. - Introduces AgentConnectStatusCommand, where Host checks whether lock for the Host is available (i.e. "previous" connect process is finished). - Implementes logic to check whether Management Server has lock against Host (exposed MySQL DB lock presence via API) - Removes synchronization on Host disconnect process, double-disconnect logic in clustered Management Server environment, added early removal from ping map (in case of combination ping timeout delay + synchronized disconnect process the Agent Manager submits more disconnect requests) - Introduces parameterized connection and status check timeouts - Implements backoff algorithm abstraction - can be used either constant backoff timeout or exponential with jitter to wait between connection Host attempts to Management Server - Implements ServerAttache to be used on the Agent side of communication (similar to Attache on Management Server side) - Enhances/Adds logs significantly to Host Agent and Agent Manager logic to trace Host connecting and disconnecting process, including ids, names, context UUIDs and timings (how much time took overall initialization/deinitialization) - Adds logs to communication between Management Servers (PDU requests) - Adds DB indexes to improve search performance, uses IDEMPOTENT_ADD_INDEX for safer DB schema updates

sureshanaparti · 2026-05-07T10:16:04Z

@blueorangutan package

blueorangutan · 2026-05-07T10:18:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-05-07T11:03:53Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17753

borisstoyanov · 2026-05-07T11:16:10Z

@blueorangutan test

blueorangutan · 2026-05-07T11:18:03Z

@borisstoyanov a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-05-07T11:42:15Z

[SF] Trillian Build Failed (tid-16039)

sureshanaparti · 2026-05-08T08:04:27Z

@blueorangutan test keepEnv

blueorangutan · 2026-05-08T08:06:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-05-08T08:30:35Z

[SF] Trillian Build Failed (tid-16049)

boring-cyborg Bot added component:agent component:api component:database component:orchestration labels Apr 15, 2026

sureshanaparti requested a review from nvazquez April 15, 2026 05:17

sureshanaparti added this to the 4.23.0 milestone Apr 15, 2026

winterhazel self-requested a review April 20, 2026 15:45

github-actions Bot added the status:has-conflicts label Apr 30, 2026

sureshanaparti force-pushed the indirect-agent-connection-improvements branch from d7a044d to d736a4a Compare April 30, 2026 06:02

github-actions Bot added status:has-conflicts and removed status:has-conflicts labels Apr 30, 2026

mprokopchuk and others added 2 commits May 7, 2026 15:45

fix test

e633b54

sureshanaparti force-pushed the indirect-agent-connection-improvements branch from d736a4a to e633b54 Compare May 7, 2026 10:15

Conversation

sureshanaparti commented Apr 15, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Apr 15, 2026

Uh oh!

sureshanaparti commented Apr 15, 2026

Uh oh!

blueorangutan commented Apr 15, 2026

Uh oh!

codecov Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Apr 15, 2026

Uh oh!

sureshanaparti commented Apr 15, 2026

Uh oh!

blueorangutan commented Apr 15, 2026

Uh oh!

blueorangutan commented Apr 15, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

borisstoyanov commented May 7, 2026

Uh oh!

blueorangutan commented May 7, 2026

Uh oh!

blueorangutan commented May 7, 2026

Uh oh!

sureshanaparti commented May 7, 2026

Uh oh!

blueorangutan commented May 7, 2026

Uh oh!

blueorangutan commented May 7, 2026

Uh oh!

borisstoyanov commented May 7, 2026

Uh oh!

blueorangutan commented May 7, 2026

Uh oh!

blueorangutan commented May 7, 2026

Uh oh!

sureshanaparti commented May 8, 2026

Uh oh!

blueorangutan commented May 8, 2026

Uh oh!

blueorangutan commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Apr 15, 2026 •

edited

Loading